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The  optimum  weighting  of  variables  to  predict  a dependent/criterion  variable 
is  an  important  problem  in  nearly  all  of  the  social  and  natural  sciences. 
Although  the  predominant  method,  multiple  regression  analynis  (MR),  yields 
optimum  weights  for  the  sample  at  hand,  these  weights  are  not  generally  optimum 
in  the  populatloi^rom  which  the  sample  was  drawn.  A method  was  developed  that 
sacrifices  some'*^edlctlon*^*ln  the  sample  at  hand  in  order  to  achieve  a more 
reliable  and  stable  predictor  composite.  The  method  developed.  Factor  Regres- 
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Intercorrelation  matrix  with  validities  in  the  diagonal  cells.  FRA  yielded 
very  stable  predictor  composites  and  weights — the  weights  themselves  varied 
less  from  sample  to  sample  than  did  MR  weights  from  the  same  samples.  These 
differences  were  marked  for  low  sample  sizes  (e.g.,  N “ 25),  regardless  of 
the  number  of  variables  In  regression.  With  regard  to  prediction,  FRA  com- 
posites were  substantially  more  valid  In  the  population  than  the  MR  composites 
based  on  the  same  samples.  The  number  of  predictors  In  the  subset  did  not 
turn  out  to  be  very  Important.  FRA  weights  based  on  samples  of  25  were  about 
as  valid  as  MR  weights  based  on  samples  of  100.  With  samples  of  200  the  two 
methods  yielded  roughly  equivalent  prediction. 
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FOREWORD 


The  early  developmental  work  of  this  new  method  was  conducted  in  support 
of  the  Interlaboratory  Independent  Research  Program.  During  FY  1976  and  7TQ, 
it  was  supported  by  the  Office  of  Naval  Research. 

Appreciation  is  expressed  to  Edward  Alf  for  his  contributions  during 
the  early  phases  of  this  effort. 


J.  J.  CLARKIN 
Commanding  Officer 


SUMMARY 


Problem 

The  optinum  weighting  of  variables  to  predict  a dependent  criterion 
variable  is  an  important  problem  in  nearly  all  of  the  social  and  natural 
sciences.  Although  the  predominant  method,  multiple  regression  analysis 
yields  optimum  weights  for  the  sample  at  hand,  these  weights  are  not 
generally  optimum  in  the  population  from  which  the  sample  was  drawn. 
However,  all  other  multivariate  methods  proposed  to  date  have  provided 
poorer  prediction  in  some  samples  and  populations. 

Obi active 

The  objective  was  to  develop  and  validate  a multivariate  method  that 
would  provide  better  prediction  weights  than  existing  methods.  The  weig 
would  be  more  stable  from  sample  to  sample,  and  would  predict  the  depend 
criterion  variable  more  accurately  in  the  population. 


Approach 


A method  was  developed  that  sacrifices  some  ''prediction"  in  the  sample 
St  hand  in  order  to  achieve  a more  reliable  and  stable  predictor  composite. 
Multiple  regression  (MR)  does  the  opposite,  assuming 

and  trustworthy  data.  The  method  developed.  Factor  Regression  Analysis  (FRA), 
is  based  on  the  first  principle  component  of  the  predictor  intercorrelation 
matrix  with  validities  in  the  diagonal  cells. 

Two  kinds  of  data  were  used  to  evaluate  FRA:  Navy  data  and  Monte  Carlo 

data.  Hundreds  of  samples  were  analyzed,  and  the  weights  were  applied  to 
i-h*i  nomilatlons  from  which  the  samples  were  dra%m.  All  predictor  subset 


Findings 


The  new  method,  FRA,  yielded  very  stable  predictor  composices  anu  wbo-ruu-- 
the  weights  themselves  varied  less  from  sample  to  sample  than  did  MR  weights 
from  the  same  samples.  These  differences  were  marked  for  low  sample 
(e.g.,  N - 25),  regardless  of  the  number  of  variables  in  regression.  For  larg< 
sample  sizes  (e.g.,  N - 200),  the  differences  were  smaller  but  still  fairly 
consistent. 

For  small  samples,  FRA  composites  were  much  more  valid  in  the  population 
than  the  MR  composites  based  on  the  same  samples.  The  number  of  predictors 
in  the  subset  did  not  turn  out  to  be  very  important.  FRA  weights  bas^ 
samples  of  25  were  about  as  valid  as  MR  weights  based  on  samples  of  100.  With 
samples  of  200  the  two  methods  yielded  roughly  equivalent  prediction. 
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Concluslona 
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The  new  method,  FRA,  Is  a very  Important  Improvement  upon  multiple  regres- 
sion for  small  samples.  In  addition,  FRA  does  not  fall  at  the  higher  samples 
sizes  as  previously  proposed  methods  have  done.  Thus,  it  can  be  used  with 
confidence  for  all  samples  sizes  as  they  occur  in  applied  settings. 
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INTRODUCTION 


Problem 

Since  1900,  It  has  been  commonly  assumed  that  multiple-regression 
weights  based  on  a sample  are  the  best  candidates  for  prediction  weights 
in  the  population  (see  Gullikson,  1950,  & Walker,  1958).  Evidence  that 
this  is  generally  not  true  for  small  samples  has  been  accumulating  for 
many  years  (e.g.,  Horst,  1941;  Guilford,  1954;  Schmidt,  1971).  Although 
several  multivariate  methods  have  been  advocated  in  recognition  of  this 
fact  (e.g.,  Horst,  1941;  Kendall,  Note  1;  Rock,  Linn,  Evans,  & Patrick, 

1970),  none  have  consistently  excelled  multiple  regression,  even  with 
rather  small  samples  (e.g.,  N^  = 50).  In  fact,  a recent  study  (Rock  et  al., 
1970)  found  that  sample  multiple-regression  weights  predicted  Y in  the 
population  better  than  did  weights  computed  by  the  other  promising  multi- 
variate methods. 

Multiple  regression  yields  a predictor  composite  that  is,  for  the 
sample  at  hand,  maximally  related  to  the  criterion.  However,  in  the  pro- 
cess, multiple  regression  violates  a principle  for  constructing  a reliable 
composite:  the  higher  the  Intercorrelation  among  the  components  of  the 

composite,  the  higher  the  reliability  of  the  composite  (Moiser,  1943). 

When  multiple  regression  computes  the  optimum  weights,  it  computes  the 
weight  for  each  predictor  while  all  other  predictors  are  held  constant. 

Thus,  each  predictor's  weight  is  based  on  its  validity  after  all  the 
criterion-related  variance  that  the  predictor  shares  with  the  other  pre- 
dictors has  been  removed.  In  effect,  each  weight  is  based  on  a residual 
matrix  or,  equivalently,  partial  correlation.  When  two  predictors  are 
highly  correlated,  the  one  with  the  larger  validity  will  tend  to  receive 
a representative  weight  and  the  other  a much  lower,  or  even  negative 
weight.  There  is  no  overlap  between  any  of  the  weights  as  far  as  account- 
ing for  criterion  variance  is  concerned.  Multiple-regression  rules  out  any 
capitalization  upon  the  redundancy  that  invariably  exists  among  the  pre- 
dictors. In  contrast,  reliable  composites  tend  to  give  great  weight  to 
intercorrelation  among  the  predictors. 

This  inconsistency  between  reliability  of  the  predictor  composite,  and 
optimum  "prediction"  in  the  sample  at  hand,  can  be  seen  in  the  formula  for 
the  standard  error  of  multiple-regression  weights.  Examination  of  that 
formula  leads  to  the  unmistakable  conclusion  that  the  higher  the  correlations 
among  the  predictors,  the  higher  the  standard  errors  of  the  weights.  This 
indicates  an  unreliable  composite  in  a situation  that  should  yield  a very 
reliable  composite.  Beaton,  Rubin,  and  Barone  (1970)  demonstrated  this 
empirically,  showing  that  the  higher  the  intercorrelation,  the  more  unstable 
the  multiple-regression  weights. 
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Another  Inadequacy  in  multiple  regression  has  been  referred  to  as  the 
"partlalllng  fallacy"  (Gordon,  1968).  As  mentioned  above,  multiple  regression 
la  based  on  partial-correlational  techniques.  It  determines  optimum  weights 
by  partlalllng — each  predictor's  optimum  weight  Is  determined  by  partlalllng 
out  association  %flth  the  other  predictors.  Unfortunately,  partial  correlations 
are  based  on  the  assumptions  that  the  variable  being  partlalled  out  contains 
no  unique  component  and  Is  measured  without  error  (Burks,  1926;  Monroe  & 

Stult,  1935;  Stephenson,  1935;  Gordon,  1968).  Few,  if  any,  psychological 
tests  and  measurements  satisfy  both  of  these  assumptions.  Unfortunately, 
no  publications  could  be  found  that  demonstrated  empirically  how  the  fal- 
lacious partial  correlations  affected,  or  did  not  affect,  multiple-regres- 
sion weights  or  their  validity. 

It  would  appear,  therefore,  from  several  standpoints,  that  multiple 
regression  does  not  give  due  consideration  to  the  unreliability  and  complexity 
of  the  measures  used  throughout  most  of  the  behavioral  and  social  sciences. 
Perhaps  compromise  between  composite  reliability  and  maximization  of  composite 
validity  In  the  sample  at  hand,  would  yield  better  weights  for  prediction  In 
the  population. 


[ Purpose 

L 

‘ The  purpose  of  this  effort  was  to  develop  and  validate  a multivariate 

method  that  would  provide  better  predictor  weights  than  existing  methods. 
These  weights  would  be  more  stable  from  sample  to  sample  and  would  predict 
I the  dependent/criterion  variable  more  accurately  in  the  population. 
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Factor  Regression  Analysis  (FRA) 

The  method  developed  Is  called  Factor  Regression  Analysis  (FRA).  Only 
the  empirical  steps  in  the  actual  procedure  will  be  presented  here.  A 
presentation  of  FRA  and  its  relationship  to  multiple  regression,  principle 
components,  and  factor  analysis  will  be  found  in  the  discussion  section. 

Starting  with  the  intercorrelation  matrix  of  predictors,  the  unities 
in  the  main  diagonal  cells  were  replaced  by  validities — each  predictor's 
validity  was  placed  in  the  predictor's  diagonal  cell.  The  first  principle 
component  of  this  matrix  was  computed  (Gulliksen,  1950,  p.  346),  and  the 
loadings  of  the  predictors  on  the  first  principle  component  were  used  as 
predictor  weights. 

Predictors  were  dropped  in  a step-wise  manner  by  omitting  the  predic- 
tor with  the  smallest  loading.  The  dropped  predictor  was  omitted  from  the 
correlation  matrix,  and  a new  principle  component  was  computed.  This  pro- 
cedure was  followed  until  only  two  predictors  remained. 

Multiple  Regression 

Step-wise  multiple  regression  was  used,  since  it  was  considered  pre- 
ferable to  the  "forward"  selection  and  the  "backward"  elimination  procedures 
(Kerllnger  & Pedhazur,  1973).  At  each  step,  all  possible  combinations,  of  a 
particular  size,  were  compared.  The  selected  subset  was  the  one  that  cor- 
related highest  with  Y. 

Test  Data 

Two  kinds  of  data  were  used  to  compare  FRA  and  multiple  regression  (MR) : 
Navy  data  and  Monte  Carlo  data.  The  Navy  data  consisted  of  16,826  records 
for  enlisted  men,  obtained  from  the  Naval  Examination  Center.  A composite 
variable.  Final  Multiple  Score  (FMS) , which  is  used  for  the  advancement 
(promotion)  decision,  was  the  criterion  variable  (Robertson,  James,  & 

Royle,  1972).  Ten  predictors  were  used:  eight  aptitude  tests  (the  Navy 

Basic  Test  Battery),  age,  and  amount  of  education  (in  years).  The  16,826 
records  were  placed  in  random  order,  and  successive  groups  were  used  as 
samples.  All  of  the  records  were  used  as  the  population. 

Monte  Carlo  data  were  generated  from  published  correlation  matrices 
(Rock  et  al.,  1970;  McCornack,  1970).  Predictors  ranged  in  number  from  11 
to  15.  A computer  program  (Moonan  & Cohver,  1973,  Note  2)  was  used  to  generate 
Monte  Carlo  samples  based  on  the  published  correlation  matrices. 

Population  Validity 

The  weights  for  each  predictor  set  or  subset  were  applied  to  the  popula- 
tion through  the  use  of  the  following  formula: 

r 
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where: 


■ sample  weight  for  the  1-th  predictor, 

^lY  * population  validity  of  the  i-th  predictor, 

Xl  “ Op/Og — the  standard  deviation  In  the  population,  over  the  standard 
deviation  In  the  sample,  and 

r^^j  = population  correlation  between  the  1-th  and  j-th  predictor. 

This  Is  called  the  "population  validity."  It  Indicates  how  well  the  sample 
weights  would  work  In  the  population.  FRA  and  MR  weights  were  evaluated 
using  this  formula. 

Computer  Program 

A computer  program  was  developed  for  the  GA1830  Computer  to  perform  the 
calculations. 
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RESULTS 


The  Intercorrelation  matrix  for  the  "population"  of  Navy  data  (l^  « 16,826) 
la  presented  in  Table  1.  Multiple-regression  weights  for  these  data  were: 


X Variable: 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

MR  Weight  : 

.15 

.08 

.12 

-.01 

.01  - 

.05 

.11 

.11 

.07 

.09 

multiple  correlation  was 

.46. 

In  the 

first 

sample  (N 

= 25), 

the  weights 

MR  Weight  : 

.10 

-.55 

.49 

.21  - 

.49 

.04 

.62 

-.04 

.28 

-.25 

FRA  Weight: 

.30 

.28 

.54 

.26 

.10 

.18 

.55 

.34 

.08 

.09 

These  weights  yielded  a population  validity  of  .24  in  the  case  of  MR,  and  .42 
in  the  case  of  FRA.  Sample  validities  for  these  two  sets  of  weights  were  .75 
and  .56,  respectively. 


Table  1 

Intercorrelation  Matrix  for  the  Navy  Data 
(N  - 16,826) 


Y 

Xi 

X2 

X3 

X4 

X5 

Xe 

X? 

Xe 

Xg 

XlO 

Y 

— 

33 

28 

26 

04 

09 

05 

34 

28 

20 

25 

Xi 

— 

56 

14 

16 

21 

30 

61 

21 

17 

38 

X2 

— 

07 

21 

09 

24 

58 

12 

19 

33 

X3 

— 

-04 

18 

02 

22 

68 

03 

00 

X4 

— 

08 

28 

15 

-07 

04 

20 

Xs 

— 

34 

18 

14 

-03 

07 

Xe 

■ 

— 

24 

-01 

02 

15 

Xy 

1 

— 

27 

19 

35 

Xe 

• 

— 

04 

05 

Xg 

— 

57 

XlO 

' 

1 

f 

— 

Note;  Decimal  points  have  been  omitted  from  correlations. 
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Table  2 presents  the  average  population  validity  In  10  samples.  This 
is  given  separately  for  four  sample  sizes  (25,  50,  100,  and  200),  as  well 
as  for  predictor  subsets  of  all  possible  sizes.  Of  most  Interest  in  the 
table  is  the  "Mean  Difference"  column.  It  shows  the  difference,  on  the 
average,  in  the  population  validities  for  MR  and  FRA  weights  given  in  the 
first  two  columns.  Each  value  refers  to  a different  predictor  subset  size 
from  2 to  10.  It  can  be  seen  that  the  FRA  weights  were  much  more  valid 
than  the  MR  weights.  Except  for  very  small  subsets,  e.g.,  two  or  three, 
it  mattered  little  how  many  predictors  were  in  the  subset. 

The  mean  differences  were  quite  large  in  many  instances,  e.g.,  15  or 
so  correlation  points.  Even  with  only  four  predictors  in  the  subset  and 
an  of  50,  the  mean  difference  was  11.  In  general,  FRA  weights  based  on 
samples  of  25  were  as  valid  as  MR  weights  based  on  samples  of  100. 

For  the  largest  samples  (N  = 200),  MR  weights  were  a little  better  fpr 
the  small  subsets  of  two  or  three  predictors.  However,  FRA  weights  achieved 
an  average  validity  of  .43  with  six  predictors,  a level  never  achieved  by 
MR  weights  for  any  subset. 

Standard  deviations  of  the  population  validities  across  10  samples  are 
presented  in  the  last  two  columns  of  Table  2.  As  can  be  seen,  the  popula- 
tion validities  differed  greatly  for  MR  but  not  for  FRA.  In  fact,  they 
were  nearly  as  consistent,  in  the  case  of  FRA,  when  ^ ■ 25  as  when  - 200. 
This  was  definitely  not  true  for  MR. 

The  standard  deviations  of  the  predictor  weights,  as  opposed  to  the 
standard  deviations  of  the  population  validities,  are  presented  in  Table  3. 
These  standard  deviations  pertain  only  to  the  weights  when  all  predictors 
were  in  regression.  The  variability  of  each  predictor's  weight  across  10 
samples  is  indicated  for  MR  and  FRA.  It  can  be  seen  that  the  FRA  weights 
were  much  more  stable  on  the  average.  This  is  especially  true  for  small 
samples.  However,  even  when  N =•  200,  the  MR  weight  for  predictors  four,  six, 
and  ten  varied  much  more  than  the  FRA  weight. 

Before  proceeding  to  other  populations,  the  selection  of  predictors  for 
inclusion  in  the  subsets  should  be  examined.  Table  4 shows  the  percentage 
of  predictors  in  common  for  the  MR  and  FRA  subsets  when  N_  = 50.  It  is  clear 
that  MR  and  FRA  did  not  use  the  same  predictors.  Results  for  the  other  sample 
sizes  were  almost  identical  and  are,  therefore,  not  given  here.  Sample  size 
was  simply  not  Important  in  this  respect.  In  no  case,  even  for  large  samples, 
was  there  complete  overlap  in  the  small  subsets.  Further,  there  were  many 
cases  in  which  only  one-third  are  common.  The  last  column  shows  the  average 
percent  for  each  subset  size. 

Three  other  "populations"  were  used  to  evaluate  FRA,  as  was  done  above 
using  Mavy  data.  There  is  one  important  difference,  however.  Instead  of 
using  whatever  data  were  available,  published  data  that  had  been  analyzed 
in  Journal  articles  were  used.  This  enabled  the  comparison  of  results,  and 
the  choice  of  widely  differing  populations. 
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Table  2 


Population  Valldltlea  of  MR  and  FRA  Conpoaicea  for  Navy  Data 


Nuabar  of  Average  Mean  Standard 

Pradictora  Population  Validity  Difference  Deviation 

In  Kegreaalon  MR  FRA  MR  FRA 


N - 25 


2 

27 

32 

05 

13 

07 

3 

26 

37 

11 

13 

04 

4 

27 

38 

11 

13 

03 

5 

25 

40 

15 

11 

03 
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Table  3 

Standard  Deviation  of  Weights  Across  10  Samples  of  Navy  Data 


Predictor 


Weights 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

N - 25 

MR 

.29 

.35 

.30 

.20 

.31 

.23 

.21 

.21 

.25 

.29 

FRA 

.19 

.12 

.20 

.15 

.05 

.06 

.08 

.12 

.08 

.21 

• 50 

MR 

.14 

.12 

.14 

.07 

.06 

.09 

.12 

.15 

.15 

.13 

FRA 

.06 

.12 

.13 

.10 

.10 

.09 

.08 

.15 

.16 

.11 

N ■ 

- 100 

MR 

.11 

.19 

.12 

.13 

.05 

.06 

.18 

.16 

.10 

FRA 

.04 

.06 

.05 

.08 

.08 

.07 

.15 

N ' 

- 200 

MR 

.06 

.09 

.10 

.10 

.03 

.10 

.07 

.08 

.08 

.11 

FRA 

.02 

.05 

.11 

.02 

.03 

.04 

.04 

.07 

.06 

.04 

Table  A 


Percentage  of  Predictors  In  Common  for  the  MR  and  FRA  Subsets 


Number  of 
Predictors 
In  Regression 

Sample 

Average 

Percentage 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

2 

50 

50 

50 

50 

50 

50 

00 

50 

50 

50 

45 

3 

67 

33 

33 

33 

33 

67 

00 

33 

67 

33 

38 

4 

75 

50 

75 

75 

50 

50 

25 

25 

50 

25 

50 

5 

60 

40 

80 

60 

40 

40 

40 

20 

60 

60 

50 

6 

67 

67 

83 

67 

50 

50 

67 

33 

50 

67 

60 

7 

71 

71 

86 

71 

71 

71 

86 

57 

71 

71 

73 

8 

75 

87 

87 

87 

75 

87 

87 

75 

75 

87 

82 

9 

89 

89 

89 

89 

89 

89 

100 

89 

89 

89 

90 

10 

100 

100 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Two  of  the  populations  used  were  analyzed  In  a journal  article  (Rock  et  al., 
1970)  that  compared  several  predictor  weighting  and  selection  techniques — In- 
cluding multiple  regression.  In  that  study,  multiple-regression  weights  were 
shown  to  be  better  than  weights  derived  by  any  of  the  other  techniques.  If 
FRA  weights  are  more  valid  on  these  data,  It  would  be  very  strong  evidence. 
Indeed,  for  the  method's  utility.  Furthermore,  the  two  populations  were 
originally  chosen  by  Rock  et  al.  (1970)  because  they  differed  greatly  In  terms 
of  level  of  validities  and  level  of  predictor  Intercorrelation. 

The  intercorrelation  matrices  of  the  two  Rock  et  al.,  (1970)  populations 
are  presented  In  Tables  5 and  6,  and  average  population  validities  for  10 
Monte  Carlo  samples  are  presented  In  Tables  7 and  8.  By  and  large,  the 
results  substantiate  the  results  for  Navy  data.  The  FRA  weights  were  much 
more  valid  for  IJ  = 25  and  nearly  Identical  to  MR  weight  validities  for  >4  « 200. 
As  In  the  case  of  Navy  data,  FRA  weights  based  on  samples  of  25  were  about  as 
valid  as  MR  weights  based  on  samples  of  100.  Even  when  the  subset  contained 
only  two  predictors,  FRA  weights  were  as  valid  as  MR  weights. 

Sample-to-sample  fluctuations  of  the  population  validities  for  FRA  were 
very  small,  as  Indicated  by  the  standard  deviations  given  In  the  last  column 
of  the  tables.  For  example,  in  Table  7,  when  IJ  • 25  and  when  12  predictors 
were  In  regression,  the  standard  deviation  of  population  validities  were  .01 
for  FRA,  compared  to  .15  for  MR.  This  is  for  an  average  validity  of  .71  for 
FRA  and  .52  for  MR.  Comparing  this  with  ^ **  100  and  12  predictors,  the 
average  population  validity  for  MR  Is  now  .71  also,  but  Its  standard  deviation 
Is  .02 — still  not  as  low  as  the  standard  deviation  for  FRA  when  N ~ 25. 
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Table  3 


r 

I 

j 

! 


Intercorrelation  Matrix  for  Rock  I 


Y 

"l 

"2 

"3 

"4 

"5 

"6 

'S 

"8 

"9 

"lO 

"ll 

"l2 

Y ~ 

55 

58 

64 

61 

56 

64 

62 

61 

40 

41 

45 

43 

^1 

— 

58 

66 

66 

66 

64 

65 

73 

49 

45 

46 

46 

— 

70 

66 

64 

65 

68 

57 

45 

52 

49 

49 

— 

79 

76 

73 

80 

65 

52 

60 

63 

57 

— 

75 

79 

80 

63 

54 

56 

59 

57 

— 

70 

76 

61 

51 

54 

58 

56 

^6 

— 

78 

65 

54 

56 

56 

56 

^7 

— 

65 

63 

67 

66 

66 

^8 

— 

47 

43 

44 

47 

— 

56 

51 

54 

^10 

— 

59 

62 

^11 

— 

55 

''12 

— 

Note:  Decimal  points  have  been  omitted  from  correlations. 
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Tabl*  8 


Population  Valldltlaa  of  MK  and  PBA  Coopoaltaa  for  lock  II 


Wmitt  of 

Anar ana 
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frodlctora 
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a - 

25 

2 
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34 

05 
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07 

3 

30 

38 
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08 

04 

4 

31 

39 

08 
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03 

5 

29 

40 

11 
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01 

6 

29 

40 

11 

07 

02 

7 

27 

40 

13 

08 

02 

8 

27 

40 

13 

09 

02 

9 

27 

40 

13 

09 

02 

10 

27 

40 

13 

09 

02 

11 

27 

40 

13 

09 

02 

12 

27 

40 

13 

09 

02 

11-50 


2 

34 

36 

02 

08 

07 

3 

34 

39 

05 

08 

05 

4 

35 

39 

04 

07 

02 

5 

35 

40 

05 

08 

01 

6 

34 

40 

06 

09 

01 

7 

35 

41 

06 

09 

01 

8 

34 

41 

07 

08 

01 

9 

35 

41 

06 

08 

01 

10 

34 

41 

07 

08 

01 

11 

34 

41 

07 

09 

01 

12 

34 

41 

07 

09 

01 

N • 100 


2 

36 

39 

03 

04 

02 

3 

36 

39 

03 

08 

01 

4 

36 

39 

03 

08 

02 

5 

37 

39 

02 

04 

01 

6 

37 

39 

02 

04 

00 

7 

38 

39 

01 

04 

00 

8 

38 

39 

01 

04 

00 

9 

38 

39 

01 

04 

00 

10 

39 

38 

-01 

04 

01 

11 

39 

38 

-01 

04 

00 

12 

39 

37 

-02 

04 

00 

a - 200 

2 

38 

39 

01 

02 

02 

3 

37 

39 

02 

02 

02 

4 

38 

39 

01 

02 

02 

5 

39 

39 

00 

02 

01 

6 

40 

39 

-01 

02 

01 

7 

40 

39 

-01 

02 

00 

1 

40 

38 

-02 

02 

00 

9 

40 

39 

-01 

02 

00 

10 

40 

38 

-02 

02 

01 

11 

39 

37 

-02 

02 

00 

12 

39 

38 

-01 

02 

00 

tiotm:  Ooelaal  poiata  bava  boon  oalttad. 


13 


Similar  results  are  presented  in  Table  8 for  the  other  population  from 
Rock  et  al.,  (1970).  FRA  performed  even  better  than  In  previous  populations 
for  small  (two  or  three)  predictor  subsets.  The  average  population  validity 
for  FRA  with  two  predictors  was  higher  than  it  was  for  MR  in  every  sample 
size.  True,  the  differences  are  small  for  the  larger  sample  sizes.  When 
N ■ 100,  however,  there  was  much  less  fluctuation  from  sample-to-sample  for 
FRA  validities.  When  N_  = 200,  the  FRA  validities  were  still  more  stable,  in 
general,  than  the  MR  validities. 

The  fourth  population — like  the  second  and  third  populations — also  had 
been  analyzed  in  a previous  journal  article — one  devoted  to  comparing  popula- 
tion validities  of  predictor  selection  techniques  (McCornack,  1970).  Table  9 
contains  the  intercorrelation  matrix,  and  Table  10,  the  comparltlve  results. 
Since  these  tables  Include  15  predictors,  the  samples  of  25  are  dangerously 
close  to  total  instability  for  multiple  regression  purposes.  However,  FRA 
performed  very  well.  Indeed.  It  gave  a high  and  very  stable  population 
validity  at  all  subset  sizes.  Its  average  population  validity  with  15  pre- 
dictors in  regression  was  .81 — 16  correlation  points  above  the  one  for  MR, 
and  Its  standard  deviation  was  only  .02  across  the  10  samples. 

With  sample  sizes  of  50,  MR  achieved  high  population  validities,  but  they 
were  less  stable  than  the  FRA  validities.  Except  for  small  subsets,  MR  and 
FRA  performed  equally  well  in  the  larger  samples.  In  the  small  subsets,  e.g., 
when  only  two  or  three  predictors  were  in  regression,  FRA  excelled.  Even 
when  ^ ■ 200,  FRA  gave  a higher  and  more  stable  population  validity  for  the 
two-predictor  subset. 

Population  validities  for  unit  weights  were  also  computed  for  all  four 
populations.  Various  studies  have  shown  that  unit  weights  often  yield  higher 
cross-validation  than  MR  weights  when  small  samples  are  used  (e.g.,  Perloff, 
1951,  Note  3;  Schmidt,  1971).  With  all  predictors  In  regression,  and  ° 25, 

Che  population  validities  for  FRA  were  larger  than  the  population  validities 
for  unit  weights — .07,  .03,  .03,  and  .26  correlation  points  in  the  four  popula- 
tions, respectively. 

Additional  analyses  were  also  conducted  in  order  to  test  the  feasibility 
and  desirability  of  using  the  second  principle  component  to  enhance  prediction. 
The  mathematical  procedure  for  combining  the  second  component  with  the  first 
component  so  as  to  optimally  predict  Y was  formulated.^  Basically,  it  in- 
volved the  computation  of  the  best  least  square  weighting  vector  for  estimating 
the  sample  Y from  the  first  two  principle  components.  This  procedure  did  not 
enhance  prediction  of  Y in  any  of  the  populations.  In  fact,  in  general  there 
was  a slight  attenuation  (.01)  of  the  prediction  provided  by  the  first  component 
alone. 


^The  author  is  indebted  to  Paul  Horst  for  his  assistance. 
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Intercorrelation  Matrix  from  McCornack 
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Note:  Decimal  points  have  been  omitted  from  correlations. 


Table  10 

Population  Validities  of  HR  and  FRA 
Compoaitsa  for  McCornack  Data 


DISCUSSION  AND  CONCLUSIONS 


Factor  Regression  Analysis  (FRA)  went  through  a long  developmental 
process.  Thus,  as  presented  above.  It  does  not  greatly  resemble  Its  earlier 
self.  Nevertheless,  Judging  from  the  reactions  of  colleagues  to  whom  the 
author  has  presented  FRA,  Its  earlier  forms  and  developments  are  very  in- 
structive and  meaningful.  Some  years  ago  the  author  had  become  convinced 
that  the  mathematical  process  called  multiple  regression  (MR)  was  far  too 
precise  for  most  of  our  samples — that  it  assumed  far  more  stability  and 
reliability  than  was  present  in  our  data.  Therefore,  it  capitalized  on 
chance  to  such  an  extent  that  weights  based  on  sample  were  likely  to  be 
very  poor  in  the  population  from  which  the  sample  was  drawn.  It  was  felt 
that  some  restraint  could  be  placed  on  the  mathematical  process  of  weight 
determination  by  eliminating  the  actual  Y scores  while  retaining  the  subjects' 
order  on  the  Y scale.  This  was  done  by  pairing  subjects  adjacent  on  the  Y 
scale — the  highest  with  the  next  highest,  the  third  highest  with  the  fourth 
highest,  etc.,  until  N/2  pairs  were  obtained. 

The  author  suspected  that  these  pairs  could  be  used  to  compute  "regres- 
sion coefficients"  for  predicting  Y,  without  directly  using  the  Y scores. 
Obviously,  the  intrapair  correlation  on  Y would  be  very  high — it  proved  to 
be  about  .99,  even  with  rather  small  samples  (e.g.,  N = 50).  (This  is  an 
intraclass  correlation — both  axes  of  the  response  surface  refer  to  the  same 
variable,  and  a plotted  point  represents  one  subject-pair.) 

What  would  the  intrapair  correlation  be  for  a given  predictor  after 
matching  on  Y?  The  author  felt  this  correlation  would  be,  to  some  extent, 
dependent  upon  the  predictor's  validity,  i.e.,  its  correlation  with  Y. 
Conversations  with  other  research  psychologists  and  statisticians,  however, 
revealed  that  the  dependency,  if  it  existed,  was  certainly  not  obvious. 
Consequently,  the  empirical  path  was  chosen  temporarily.  A small  subsample 
from  a large  sample  of  Navy  data  was  used.  All  intrapair  correlations  were 
computed.  The  correlations  were  generally  much  smaller  than  the  respective 
validities,  but  were  rather  obviously  correlated  with  them.  So,  the  pos- 
sibility still  remained  that  there  was  some  dependency. 

Simultaneously,  a computer  program  was  written  that  would  compute  an 
Intrapalr  correlation  for  all  the  predictors  combined,  weighting  them  so  as 
to  maximize  this  correlation.  (One  can  conceive  of  this  as  an  "Intraclass 
canonical"  correlation — a battery  of  variables  is  optimally  weighted  to  cor- 
relate with  Itself,  each  plotted  point  of  the  response  surface  representing 
a subject  pair.)  These  weights  were  then  applied  to  the  large  sample  from 
which  the  subsample  was  drawn.  It  was  discovered  that  they  predicted  Y quite 
well— better  than  the  multiple-regression  weights  from  the  same  subsample. 

The  computer  program  Is  based  on  Thomson's  application  (1940)  of  Uottelllng's 
"most  predictable  criterion"  solution  (1935).  Thomson  showed  how  one  could 
obtain  the  composite  that  Is  maximally  reliable  by  computing  the  first 
principle  component  of  a correlation  matrix  with  reliabilities  In  the  diagonal. 

In  the  present  case,  the  correlation  matrix  for  the  predictors  was  used,  with 
each  predictor's  diagonal  cell  containing  Its  Intrapalr  correlation.  The  first 
principle  component  of  this  correlation  matrix  yielded  the  weights  that  proved 
to  predict  Y In  the  population  so  well. 
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Meanwhile,  a search  of  the  literature  revealed  an  article  by  McNemar 
(1940)  which  indicated  that  the  expected  value  of  the  Intrapair  correla- 
tion for  a given  predictor  might  be  its  validity  squared.  A mathematical 
proof  was  developed  and  is  presented  in  the  appendix.  Armed  with  this 
proof,  it  was  no  longer  necessary  to  pair-off  the  subjects  on  the  basis 
of  their  Y scores.  The  computer  program  was  simply  modified  to  use  the 
squared  validity  instead  of  the  intrapair  correlation,  in  the  respective 
diagonal  cell.  Subsequent  experience  revealed  that  the  validity  itself 
worked  as  well,  or  even  better  for  some  populations,  than  the  validity 
squared. 

Thus,  the  correlation  matrix  used  in  the  computer  program  is  the 
predictor  matrix,  with  each  predictor's  validity  in  the  diagonal  cell  for 
that  predictor.  Each  and  every  element  in  the  matrix  is  based  on  the  entire 
sample  of  N subjects.  The  first-principle  component  of  this  matrix  yields 
the  FRA  weights.  This  component  has  the  greatest  variance  th  is  possible, 
representing  the  best-fitting  line  through  the  predictor  space  after  the 
predictor  scales  have  been  transformed  by  their  validities.  Two  predictors 
that  are  highly  correlated  can,  and  often  do,  have  the  same  weight.  This 
enhances  the  reliability  of  the  composites. 

It  may  be  of  interest  to  compare  FRA  and  MR  in  matrix-algebra  form. 
Weight  computation  involves  the  solving  of  the  Lagrange  multiplier,  X, 
through  a set  of  simultaneous  linear  equations  of  the  form; 

(r'R"^r  - Xl]  = 0 
in  the  case  of  MR,  and 

|r"^c  - XI]  =0 


in  the  case  of  FRA,  where 

r is  the  vector  of  validities, 

R is  the  intercorrelation  matrix  of  predictors  with 
ones  in  the  main  diagonal, 

I Is  the  identity  matrix,  and 

C is  the  intercorrelation  matrix  of  predictors  with 
validities  in  the  main  diagonal. 

The  equation  for  FRA  yields  predictor  weights  that  maximize  the  expected 
value  of  the  Intrapalr  correlation  discussed  above. 

It  can  be  seen  that  the  equations  for  MR  and  FRA  differ  only  in  the 
first  term.  In  the  case  of  MR,  R-inverse  is  pre-  and  postmultlplled  by  the 
vector  of  validities.  And  in  the  case  of  FRA,  it  Is  postmultlplled  by  C, 
which  Is  the  same  as  R except  for  the  diagonal  elements — unities  In  R,  and 
validities  In  C.  It  Is  of  Interest  to  note  that  the  FRA  equation  Is  equivalent 
to  the  one  used  by  Thomson  (1940)  to  maximize  the  reliability  of  a composite, 
except  that  the  diagonal  elements  of  matrix  C were  the  reliabilities  of  the 
respective  predictors  (see  Gulliksen,  1950,  p.  346). 
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For  completeness,  we  should  note  that  the  comparable  equation  for  factor 
analysis  Is  |C  - Alj  = 0 where  C is  the  intercorrelation  matrix  with  com- 
munalitles  Jn  the  diagonal.  When  reliabilities  are  used,  as  they  often  are 
as  communallty  estimates,  the  C matrix  is  the  same  as  the  matrix  used  by 
Thomson  (1940).  The  remaining  difference  between  factor  analysis  and  maximiz- 
ing the  reliability  of  a composite  is  that  C is  multiplied  by  R-inverse 
in  the  latter  case. 

The  new  method  is  a very  important  improvement  upon  multiple  regression 
for  small  samples.  In  addition,  FRA  does  not  fail  at  the  higher  samples 
sizes  as  previously  proposed  methods  have  done.  Thus,  it  can  be  used  with 
confidence  for  all  samples  sizes  as  they  occur  in  applied  settings. 
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APPENDIX 


PROOF  THAT  THE  VALIDITY  SQUARED  IS  THE  EXPECTED  VALUE  OF  THE 
CORRELATION  BETWEEN  TWO  SUBSETS  MATCHED  ON  THE  CRITERION 


PROOF  THAT  THE  VALIDITY  SQUARED  IS  THE  EXPECTED  VALUE  OF  THE 
CORRELATION  BETWEEN  TWO  SUBSETS  MATCHED  ON  THE  CRITERION 


The  sample  members  are  ranked  on  the  basis  of  their  criterion  (Y)  scores. 
Then,  the  sample  member  with  the  highest  Y score  is  paired  with  the  second 
highest,  the  third  highest  is  paired  with  the  fourth  highest,  etc.  In  this 
way,  the  sample  is  divided  into  two  sets,  A and  B,  which  correlate  as  high 
as  possible  on  the  criterion,  Y.  Set  A contains  all  the  odd  rank  numbers 
and  set  B contains  all  the  even  rank  numbers. 

If  we  assume  standard  scores,  the  variance  of  the  differences  on  a 
predictor,  X,  is 


(A-B) 


" = °a'  °b'  - ^’^abVb* 


where  r^  is  the  correlation  between  sets  A and  B on  X. 

But  since  “ ^^B^  ~ 1-0, 

Now,  if  we  return  to  the  original  bivariate  distribution  X and  Y for 
this  sample,  the  standard  error  of  estimate  is 

°YX  ° °y'^”'^YX^’ 


and  for  a given  Y score 


0 ^ = l~r  2. 

X-Y  *^YX 


For  the  same  Y score,  the  variance  of  the  difference  between  the  pair  groups, 
sets  A and  B,  is 


(A-B) 


°a"  + 0^2  + 0, 


where  the  zero  arises  from  the  fact  that  Er  * 0 for  a given  Y score. 

AB 


But  since,  for  a given  Y score. 


EOa  = EOg  = Ec^, 


Equation  5 can  be  written 


Ea  ^ * 2n  ^ 

‘•°(A-B)  “^^X-Y 


or,  following  Equation  4, 


^°(A-B)^  “ ad-r^jj^). 


A-1 


Notice  that  Equations  2 and  8 are  both  based  on  a conceptually  fixed 
Y.  In  Equation  2,  Y was  fixed  for  each  difference  by  matching.  In  Equa- 
tion 8,  Y was  fixed  statistically.  Therefore,  Equations  2 and  8 are 
equivalent,  and 


Thus,  the  validity  squared  is  the  expected  value  of  the  correlation  between 
two  subsets  matched  on  the  criterion. 
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